Unit irregex

This library unit provides support for regular expressions, using the powerful irregex regular expression engine by Alex Shinn. It supports both POSIX syntax with various (irregular) PCRE extensions, as well as SCSH's SRE syntax, with various aliases for commonly used patterns. DFA matching is used when possible, otherwise a closure-compiled NFA approach is used. Matching may be performed over standard Scheme strings, or over arbitrarily chunked streams of strings.

On systems that support dynamic loading, the irregex unit can be made available in the Chicken interpreter (csi) by entering

(require-extension irregex)

Specification

Procedures

irregex
string->irregex
sre->irregex
(irregex <posix-string-or-sre> [<options> ...]) procedure
(string->irregex <posix-string> [<options> ...]) procedure
(sre->irregex <sre> [<options> ...]) procedure

Compiles a regular expression from either a POSIX-style regular expression string (with most PCRE extensions) or an SCSH-style SRE. There is no (rx ...) syntax - just use normal Scheme lists, with quasiquote if you like.

Technically a string by itself could be considered a valid (though rather silly) SRE, so if you want to just match a literal string you should use something like (irregex `(: ,str)), or use the explicit (sre->irregex str).

The options are a list of any of the following symbols:

'i, 'case-insensitive
match case-insensitively
'm, 'multi-line
treat string as multiple lines (effects ^ and $)
's, 'single-line
treat string as a single line (. can match newline)
'utf8
utf8-mode (assumes strings are byte-strings)
'fast
try to optimize the regular expression
'small
try to compile a smaller regular expression
'backtrack
enforce a backtracking implementation

The 'fast and 'small options are heuristic guidelines and will not necessarily make the compiled expression faster or smaller.

string->sre
maybe-string->sre
(string->sre <str>) procedure
(maybe-string->sre <obj>) procedure

For backwards compatibility, procedures to convert a POSIX string into an SRE.

maybe-string->sre does the same thing, but only if the argument is a string, otherwise it assumes <obj> is an SRE and returns it as-is. This is useful when you want to provide an API that allows either a POSIX string or SRE (like irregex or irregex-search below) - it ensures the result is an SRE.

irregex?
(irregex? <obj>) procedure

Returns #t iff the object is a regular expression.

Searches for any instances of the pattern <irx> (a POSIX string, SRE sexp, or pre-compiled regular expression) in <str>, optionally between the given range. If a match is found, returns a match object, otherwise returns #f.

Match objects can be used to query the original range of the string or its submatches using the irregex-match-* procedures below.

Examples:

(irregex-search "foobar" "abcFOOBARdef") => #f

(irregex-search (irregex "foobar" 'i) "abcFOOBARdef") => #<match>

(irregex-search '(w/nocase "foobar") "abcFOOBARdef") => #<match>

Note, the actual match result is represented by a vector in the default implementation. Throughout this manual, we'll just write #<match> to show that a successful match was returned when the details are not important.

Matching follows the POSIX leftmost, longest semantics, when searching. That is, of all possible matches in the string, irregex-search will return the match at the first position (leftmost). If multiple matches are possible from that same first position, the longest match is returned.

irregex-match
(irregex-match <irx> <str> [<start> <end>]) procedure

Like irregex-search, but performs an anchored match against the beginning and end of the substring specified by <start> and <end>, without searching.

Examples:

(irregex-match '(w/nocase "foobar") "abcFOOBARdef") => #f

(irregex-match '(w/nocase "foobar") "FOOBAR") => #<match>
irregex-match-data?
(irregex-match-data? <obj>) procedure

Returns #t iff the object is a successful match result from irregex-search or irregex-match.

irregex-num-submatches
irregex-match-num-submatches
(irregex-num-submatches <irx>) procedure
(irregex-match-num-submatches <match>) procedure

Returns the number of numbered submatches that are defined in the irregex or match object.

irregex-names
irregex-match-names
(irregex-names <irx>) procedure
(irregex-match-names <match>) procedure

Returns an association list of named submatches that are defined in the irregex or match object. The car of each item in this list is the name of a submatch, the cdr of each item is the numerical submatch corresponding to this name. If a named submatch occurs multiple times in the irregex, it will also occur multiple times in this list.

irregex-match-valid-index?
(irregex-match-valid-index? <match> <index-or-name>) procedure

Returns #t iff the index-or-name named submatch or index is defined in the match object.

irregex-match-substring
irregex-match-start-index
irregex-match-end-index
(irregex-match-substring <match> [<index-or-name>]) procedure
(irregex-match-start-index <match> [<index-or-name>]) procedure
(irregex-match-end-index <match> [<index-or-name>]) procedure

Fetches the matched substring (or its start or end offset) at the given submatch index, or named submatch. The entire match is index 0, the first 1, etc. The default is index 0.

irregex-match-subchunk
(irregex-match-subchunk <match> [<index-or-name>]) procedure

Generates a chunked data-type for the given match item, of the same type as the underlying chunk type (see Chunked String Matching below). This is only available if the chunk type specifies the get-subchunk API, otherwise an error is raised.

irregex-replace
irregex-replace/all
(irregex-replace <irx> <str> [<replacements> ...]) procedure
(irregex-replace/all <irx> <str> [<replacements> ...]) procedure

Matches a pattern in a string, and replaces it with a (possibly empty) list of substitutions. Each <replacement> can be either a string literal, a numeric index, a symbol (as a named submatch), or a procedure which takes one argument (the match object) and returns a string.

Examples:

(irregex-replace "[aeiou]" "hello world" "*") => "h*llo world"

(irregex-replace/all "[aeiou]" "hello world" "*") => "h*ll* w*rld"
irregex-split
irregex-extract
(irregex-split <irx> <str> [<start> <end>]) procedure
(irregex-extract <irx> <str> [<start> <end>]) procedure

irregex-split splits the string <str> into substrings divided by the pattern in <irx>. irregex-extract does the opposite, returning a list of each instance of the pattern matched disregarding the substrings in between.

irregex-fold
(irregex-fold <irx> <kons> <knil> <str> [<finish> <start> <end>]) procedure

This performs a fold operation over every non-overlapping place <irx> occurs in the string str.

The <kons> procedure takes the following signature:

(<kons> <from-index> <match> <seed>)

where <from-index> is the index from where we started searching (initially <start> and thereafter the end index of the last match), <match> is the resulting match-data object, and <seed> is the accumulated fold result starting with <knil>.

The rationale for providing the <from-index> (which is not provided in the SCSH regexp-fold utility), is because this information is useful (e.g. for extracting the unmatched portion of the string before the current match, as needed in irregex-replace), and not otherwise directly accessible.

The optional <finish> takes two arguments:

(<finish> <from-index> <seed>)

which simiarly allows you to pick up the unmatched tail of the string, and defaults to just returning the <seed>.

<start> and <end> are numeric indices letting you specify the boundaries of the string on which you want to fold.

To extract all instances of a match out of a string, you can use

(map irregex-match-substring
     (irregex-fold <irx>
                   (lambda (i m s) (cons m s))
		   '()
		   <str>
		   (lambda (i s) (reverse s))))

Extended SRE Syntax

Irregex provides the first native implementation of SREs (Scheme Regular Expressions), and includes many extensions necessary both for minimal POSIX compatibility, as well as for modern extensions found in libraries such as PCRE.

The following table summarizes the SRE syntax, with detailed explanations following.

 ;; basic patterns
 <string>                          ; literal string
 (seq <sre> ...)                   ; sequence
 (: <sre> ...)
 (or <sre> ...)                    ; alternation
 
 ;; optional/multiple patterns
 (? <sre> ...)                     ; 0 or 1 matches
 (* <sre> ...)                     ; 0 or more matches
 (+ <sre> ...)                     ; 1 or more matches
 (= <n> <sre> ...)                 ; exactly <n> matches
 (>= <n> <sre> ...)                ; <n> or more matches
 (** <from> <to> <sre> ...)        ; <n> to <m> matches
 (?? <sre> ...)                    ; non-greedy (non-greedy) pattern: (0 or 1)
 (*? <sre> ...)                    ; non-greedy kleene star
 (**? <from> <to> <sre> ...)       ; non-greedy range
 
 ;; submatch patterns
 (submatch <sre> ...)              ; numbered submatch
 ($ <sre> ...)
 (submatch-named <name> <sre> ...) ; named submatch
 (=> <name> <sre> ...)
 (backref <n-or-name>)             ; match a previous submatch
 
 ;; toggling case-sensitivity
 (w/case <sre> ...)                ; enclosed <sre>s are case-sensitive
 (w/nocase <sre> ...)              ; enclosed <sre>s are case-insensitive
 
 ;; character sets
 <char>                            ; singleton char set
 (<string>)                        ; set of chars
 (or